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MEMORANDUM 
SUBJECT: Intelligence Analysis Quality 


A memorandum on managing the improvement of intelligence analysis 
asserted that defining good intelligence analysis was a necessary step 
in the management process. The attached rambling essay (of which I'm 
not particularly proud) is a first cut at such a definition. 


It will not satisfy anyone, but those who have analyzed successfully 
will see some of their instincts reflected. It is, however, abstract. 
I ask the forbearance of those who cannot provide their own examples to 
support the generalizations and ask them to accept on faith for a time 
that each point speaks to a real and important problem in intelligence 
analysis. Examples can be provided, but at considerable cost in length 
and classification. For the moment a more or less philosophical discourse 
is-needed. If we are impatient with this and so limit outselves to 
"real" problems like, "How do we fix our assessment of Taoist resurgence?" 
or "Who will make up the next class of our analytic methods course?" we 
doom ourselves to inefficient and often counterproductive gropping. 


If we are to ever get far enough ahead of the problem of improving 
intelligence to control it instead of having the problem drive us, we 
must step back, decide what we want to achieve (which the attached paper 
addresses in part), then decide how to work toward that (which the 
memorandum on management addresses in part), and finally set out to do 
it, necessarily piecemeal. 


The attached paper is intended to serve as a target for attacks. 
Therefore, ample space is left throughout for marginal notes. Hopefully 
it will be circulated for comment and criticism, e.g. to the NIOs and 
Office Directors. One could then accumulate comments and criticisms, 
discuss the issues revealed by them, and revise the paper. Some such 
process is needed if this is to be more than an academic exercise because, 
if any program is to succeed in improving intelligence analysis, it must 
fully recognize that we are trying to change human beings. Proclamations 
don't change people. People change only if they are persuaded and 
people are persuaded most readily if they are genuine participants in 
defining goals and how to achieve them. 


THIS MEMORANDUM IS UNCLASSIFIED 
UPON REMOVAL OF 
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Some copies of this memorandum will have attached copies of an 
interagency intelligence memorandum to support the premise that the 
ideas expressed in the paper are not pie-in-the-sky. If this IIM could 
be produced in the absence of clear guidance, we can reasonably expect 
that much more and better analyses can be produced if sufficently explicit 
guidance existed, and was accepted. 


Pleas for a philosophical discourse notwithstanding a reasonable 
question for a hard-nosed manager to ask is, "If I believe all this, 
what do I do?" Several answers are suggested in the memorandum on 
management. Some answers are alluded to in the attached paper. No 
answer is given either place with sufficient specificity to be implemented 
straightaway. This is because the underpinning of both papers is that 
successful solutions can only be devised by those who must implement 
them. Therefore, the papers only sketch what such solutions might look 
like. 


However, to close on a concrete note one can envision a group of 
managers and analysts debating the proposition (made in the attached 
paper) that review processes would be more efficient if they operated on 
outlines rather than finished essays and concluding that the following 
procedural rules should be implemented. 


-Each analysis will be prepared in an outline format. 


-After the analyst has completed the outline no editing 
is permitted until the review process is complete. 


-All reviewers must provide marked up copies of the basic 
analysis. The rationale for each indicated change must 
be provided. 


-The final review authority will examine the basic analysis 
together with the comments of intermediate reviewers and 
will indicate which changes are to be incorporated. 


-Then the analyst incorporates all authorized changes 
into the final analytic paper which becomes the official 
record of the analysis. 


-Thereafter an editor writes essays in various forms, 
styles and lengths (subject to the analyst's review) 
to satisfy various consumer needs. 


One could imagine this group arguing that, no matter how stilted this 
seems, we must cut the Gordian Knot if we are to hope to abolish the 
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endless nit-picking and wordsmithing that now characterize our review 
processes and that stifle creativity and bog down production. Then 
again, the group might produce a totally different plan. The one thing 
that can be said with confidence is that if they themselves produce a 
plan that addresses the problems, it will be better as a practical 
matter than anything I could conjure. 


Attachments 


os 
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INTELLIGENCE ANALYSIS QUALITY 


"Let these examples also serve to inform the 
politician of vast projects that the human mind 
can never penetrate however extensive it may be, 
because their minute combinations must be developed 
in order to foresee or regulate events that depend 
on future contingencies. We can explain past 
incidents clearly, for their causes are now dis- 
covered, but we always deceive ourselves about the 
future which is concealed by secondary causes from 
our rash and prying inspection. That the expectations 
of politicians should be disappointed is not a 
phenomenon peculiar to the present age. It has 
been the same during all the ages in which 
human ambition gave birth to grand projects." 


Fredrick, The History of the Seven 


Years War, I, viti-ix, II, 369 


Fredrick the Great of Prussia serves to start 
us on the first task, defining what good intelligence 
analysis is not. The second task, defining what it is, 
will follow and will be harder. 


WHAT GOOD INTELLIGENCE ANALYSIS IS NOT 


Fredrick pointed out that good intelligence 
analysis is not the same as consumer satisfaction. 
Consumers haven't changed in two hundred-odd years; 
they'll never be satisfied. Failures will be heralded 
and successes taken for granted, the latter because (at 
least in part) the timely anticipation of events allows 
a decisive policymaker to preempt them. This is not to 
argue for ignoring serious policymaker critiques of 
intelligence; it does argue that each critique should 
be judged on its own merits. Also it argues more 
generally that good press will not be ours. 


Second, good intelligence analysis need not mean 
good intelligence. Intelligence consists broadly 
of three phases, collection, analysis and dissemination. 
For intelligence to be successful all three steps must 
be taken; a misstep anywhere can cause failure. Absent 
the basic information, the most finely tuned analytic 
machinery cannot produce useful intelligence. Absent 
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the ability to provide assessments in forms intelligible 
and persuasive to the people who should act, the most 
insightful analysis is worthless. In any case, we will 
not address here questions of collection and dissemination 
in spite of the fact that they are important, we will 
limit ourselves to the middle stage. 


Next, when our microscope focuses on the middle 
stage we find that there are finer gradations in the 
process of intelligence analysis itself. We also see that 
intelligence analysis 7s a seamless web so that any cut 
for purposes of taxonomy does violence to the whole. 
Nonetheless for our purposes here it will be useful to 
cut the web into four sections that we may call the 
functions of intelligence and describe by the following 
catch phrases: 


FACTS & FIGURES: Who's where? When does how 
much rain fall? How big is it? How much does it 
cost? etc. 


RECKONING & REPORTING: What happened yesterday or 
did nothing happen? What is the military capability 
of an assembly of men and equipment? What is the 
productive capacity of an assembly of men and 
machines? etc. 


PREDICTION & PROGNOSES: What will happen tomorrow 
or next year? What are the critical factors 
influencing developments? What are the key 
uncertainties? etc. 


WATCH & WARD: Among all the scenarios that might 
unfold, which ones are both sufficiently likely 
and sufficiently important to the US to merit 
special attention? 


Each of these functions, make somewhat different demands 
on the analyst. Here we will not attempt to address all 
of these demands, rather we will concentrate on the 
demands of the PREDICTION & PROGNOSES function with only 
peripheral mention of the others. 
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Fourth, we will ignore questions of relevance even 
though it is clear that when all is said and done superb 
analyses of irrelevant problems are not better than shoddy 
analyses of urgent problems. Ignoring relevance only 
recognizes that we cannot address all of the problems of 
the world at once. 


Fifth and finally, we should distinguish between 
intelligence officers as “analysts” and as "seers." We 
are not talking here about the latter function, valuable 
as it may be. In discussions of these matters there is a 
tension, usually not articulated or even recognized, between 
two views. On the one hand some think of good intelligence 
as the result of certain methods of reasoning or information 
processing which if properly applied to any situation will 
produce truth. Others think that truth emerges from a 
mental process which is too subtle and complex to be 
codified so it can be achieved only by long and intimate 
involvement with the subject area. Reality surely 
lies in the middle. I know that years at sea gives a ship's 
captain a "feel" for his vessel so that he can often sense 
when things are not right, but it is only the ability to 
mentally break down the myriad of sights, sounds, smells, or 
their absence into mechanical, electrical and electronic 
systems (i.e. analysis) that allows him to convert that 
sense into a program of troubleshooting that will guide his 
crew to a defective diode or a stuck valve which can then 
be repaired. A similar, but more complex situation, must 
exist when one is trying to comprehend the workings of a 
political and economic system in another culture. And 
success can only come from a proper, and perhaps fortuitous, 
marriage of intuition and science. 


We must and do try to emerse people so fully in the 
culture of another country that they will be able to sense 
what is going to happen even if they cannot articulate why. 
This is an essential objective of the United States 
Intelligence Community, and while it is not the subject of 
our investigation here, two points about it are relevant. 
First, we seek for seers most urgently in political analysis 
because this is the area which we understand least; we 
intuitively recognize that we have no immediate hope of 
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modeling how the Lower Slobovians operate, but we trust that 
the bright people we emerse in the culture of Lower Slobovia 
for years will assimilate it in some undefined way and 

will thereafter be able to predict the course of 

Lower Slobovian politics. The second relevant point follows 
from the first. That is, compared to the paucity and 
inadequacy of the criteria for judging analysis, we have 

no criterion for judging prescience. So in the end 

Pinning our hopes on area specialists is an act of faith. 


WHAT IS INTELLIGENCE ANALYSIS? 


Having stated several things intelligence analysis is 
not, we must define what we mean by analysis in general 
and intelligence analysis in particular. One philosopher 
might define analysis as the separation of a complex whole 
into its constituent parts in order to study them and 
their interconnections individually. Another might, with 
equal support in philosophical literature, define analysis 
as testing the validity of an hypothesis by tracing out its 
implications; its validity is tentatively accepted with 
increasing confidence as more and more implications are 
traced out and matched with reality, but as soon as one 
falsehood is found the hypothesis is declared invalid. 
Each of these views captures some of what we mean eyen 
though we do not and never have used the term "analysis" 
rigorously in the Intelligence Community. Indeed, in some 
cases we have used "analysis" as a synonym for "writing 
style"; this clearly has been an inappropriate use of the 
term, but even so there remains ample room for differing 
usages. 


Having said this and further having accepted for 
purposes of discussion some combination of the two standard 
philosophical definitions of analysis, a reasonable question 
is, "What sets intelligence analysis apart from analysis 
in general?" The quick answer is, "Not very much." This 
is valuable because it suggests that we do not have to 
confine ourselves to post mortems of intelligence failures 
or even studies of intelligence successes. We can learn 
from the experiences of all the sciences and humanities. 


However, the quick answer deserves amplication lest 
we ignore two critical points that distinguishes 
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intelligence analysis from analysis in general. First, 
intelligence is different in the degree to which it 
deals with information that belongs to knowledgeable 
thinking people who have reason to conceal or distort 
it. Therefore, far more than even social scientists, 
intelligence analysts must treat their raw material 

with a skepticism just short of paranoia. Second, 
intelligence analysis, like other relevant analysis in 
government and business, often must meet deadlines. 

Such deadlines rarely exist to the same degree in the 
academic world so the writings in the academic literature 
on how to do good analyses should be taken with a grain 
of salt. The fact not recognized in some of this 
literature is that in the real world, “the excellent is 
the enemy of the good," if being excellent 

instead of just good in an analysis means that decisions 
will have been made before the policymakers see the 
excellent analysis or if it means that other areas are 
being neglected while fine-tuning one. 


We now turn to the subject at hand, "What constitutes 
good analysis?" with a final warning. Our discussion 
will be disappointing because there's not very much 
definitive to be said. (That's why it's taking so long 
to say it!) We need to admit at the outset that quality 
in intelligence analysis is fundamentally an aesthetic 
judgment. We cannot let this be an excuse for doing 
nothing, but we should not hoist ourselves on our own 
yardstick just for the sake of having some formula or 
code that looks scientific. We will never have an 
explicit quantitative measure of analytic quality, but 
we can and must have explicit qualitative statements 
of our aesthetic criteria. The remainder of this paper 
is an attempt to state them in the spirit of providing 
targets at which to shoot rather than of carving 
scriptures on tablets. So we really are talking about some 
desirable features of analysis rather than defining 
good analysis. 


For purposes of discussing analysis we will break 
it into half a dozen parts which can be called: 
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Models - the framework within which we 
view the part of the world under study. 


Assumptions - those things which are taken 
for granted. 


Hypotheses - things we don't take for 
granted and, indeed, among which we want to 
be able to judge. 


Facts - those things that we see as 
relevant to distinguishing among hypotheses 
and as bearing on our comfort with our 
assumptions and models. 


Reasoning - the process by which we combine 
assumptions and facts within our models to 
distinguish between our hypotheses. 


Conclusions - our assessments of the 
hypotheses under consideration. 


As we talk about each of these in turn we can describe 
a number of pitfalls that have been discovered, usually 
inadvertently and always painfully, by analysts down 
through the ages. And having done so, we will conclude, 
not surprisingly, that good analysis consists of having 
handled all of these six components of analysis well. 


THE COMPONENTS OF ANALYSIS 


Models are the central feature of good analysis. 
They are the bones on which the flesh can be hung. 
The word "model" is deliberately used recognizing that 
it curls many a humanist lip in disdain. This is 
because the issue is fundamental and cannot be evaded 
if there is to be any hope of dramatically improving the 
quality of intelligence analysis. Whatever the unpleasant 
connotations of being too mechanistic or too systems 
analytic, the issue must be faced squarely. Until we 
require ourselves to lay out explicitly our understanding 
of how the systems we study work, we have no hope of 
improving our institutional ability to understand year by 
year. Lacking models because there is nothing sufficiently 
solid to criticize, correct, restructure and observe, i.e. 
nothing concrete enough to grasp, and change or 
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to hold fixed to see if it works. On occasion I have 
worked with analysts of Strange cultures over a period 
of months during which time we radically altered our 
view of how people in that culture function at least 
three times, sometimes passing through old models 
without fully realizing it. Only when time permitted 
digging through the records of old work did our 
oscillations become apparent. Without explicit mental 
models intellectually we are like ants riding a 
pendulum; we think we're traveling, but we're only 
oscillating in minor intellectual fads. 


What is a model? It is an explicit statement 
of the process at work. It need not be a huge 
mathematically derived computer-based thing, although 
in some cases (such as understanding missile performance) 
that is appropriate. A model may also, for example, be a seminal 
paper on how the Saudi government and royal family 
interact, or it may be anly an outline. The point is 
that when the model is explicit we can critique it and 
improve. We can go back to previous work and say, 
"These developments are not consistent with what we 
said." We can then either modify the model or prove 
that the developments are, after all, consistent. But 
had the original work simply laid out the Delphic 
prophecy that "The Saudi royal family will see its 
future as intimately tied to developments in the Arab 
world and will react so as to protect its fundamental 
interests"...How many times have we read that?...we 
couldn't critique or improve on it. We couldn't eyen 
get hold of it! 


In addition to being explicit, what should a model 
be? It should be elegant! Elegance, that is, in the 
sense of the mathematicians to whom elegance means the 
proper balance between brevity and coverage. A model 
Should be brief and yet capture the essence of reality. 
Thus the simplest explanation need not be elegant; it 
may be merely an appeal to a slogan, e.g. explaining 
French foreign policy as arising from a French vision of 
grandeur. On the other hand, convoluted arguments are 
noxious even if they are carefully crafted and succeed 
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"in explaining reality. Copernican astronomical theory 
triumphed over the Ptolemaic not because it predicted 
the movement of celestial bodies more accurately, but 
because it was simpler and yet still predicted. 


If one believes this about models, what does one 
do? First one doesn't worry very much about OSI, OW! 
or even OSR whose people, by and large, have grown up 
acclimated to the use of models. One worries 
moderately about OER. One worries a lot about OPA where 
the whole concept of formal (i.e. explicit) mental models 
is anethema. In this environment a right first step 
is to require that everything must be outlined instead 
of written in the customary essay format. The outline 
provides a better framework for evaluation than the 
essay where verbage and adverbage can conceal a host of 
sins. If the review process operated on outlines 
(with the essays being written only after the substantive 
analytical work and review was done) it would be much 
more productive. 


Models play another role when fleshed out with 
assumptions, hypotheses, facts, reasoning and conclusions. 
Typically we become more confident of a theory or 
model as more and more facts accumulate in support of 
it and we reject it when too many "facts" are inconsistent 
with it. A model, by being explicit, gives us the ability 
to do more than simply wait helplessly to see 
what facts fate dumps in our bucket. It allows us to 
conceptually vary the components of the model one hy one 
to purposefully see if our theory is adequate. A 
particularly fortuitous combination of insight and 
circumstances in this exercise will occasionally allow 
us to formulate what some philosophers of science call 
the crucial experiment. In an intelligence officer's 
terms this means the ability to direct collectors in 
search of those crucial pieces of information that will 
allow us to determine if our whole theory is supportable. 
More often the use of explicit models will not lead us to 
such grandiose heights, but it will often give us the 
ability to guide the collectors to more fertile fields 
than they would otherwise cultivate. 
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A final comment on models risks taking this 
discussion far into esoteric areas. It is that the 
explicit statement of the relations which are assumed 
to exist in our subject area allows us to consider in 
an orderly fashion what things are constant and hence 
reliable and what things are variable. The degree to 
which we can do this will be the major determinant 
of our predictive power. For example, the reason that 
physics theories generally predict well while 
economic theories predict poorly is that physical models 
contain "conservation laws" at critical points while 
economic models do not. For example, Newtonian physics 
is largely based on the conservation of momentum and 
energy while microeconomic theory is based on the 
concept of individual economic wants or values which 
do not obey conservation laws, i.e. a person's desire 
for a good is not invariant, but increases or decreases 
in anticipation of future price changes. 

The existence of this difference between physics and economics 
would not even be apparent to us unless the two sciences 
had laid out their respective theories explicitly so that 
we could study the differences between them and recognize 
in both cases the limitations on prediction that are 
implicit in the strength of their conservation laws. Few 
intelligence models can include conservation laws as strong 
as those in economics, let alone those in physics. But 

the explicit enunciation of how we think things work will 
allow us to intelligently judge the circumstances under 
which habits will remain operative, i.e. crude 

conservation laws will be obeyed. Thus to say, "the 
Vietnamese are bloody minded" is useful only if it defines 
how the senior decisionmakers in the Vietnam government 
will respond to various pressures and defines the limits 
within which those pressures may vary without causing a 
fundamental change. 


Assumptions. Analysis in any field must proceed 
on the basis of assumptions. In fact, to a large extent, 
the degree to which a set of assumptions commonly held 
by all practioners in a field are adequate to provide 
the basis for fruitful work is a measure of the advance- 
ment of the field. The two major requirements we should 
impose on our assumptions is that they be explicit and 
that they be consistent across our hypotheses. 
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A major defect of much intelligence analysis is that 
too many assumptions are implicit. As a consequence 
errors creep into the process. Consider, for example, 
assumptions about US policy. Intelligence analysts 
read the newspapers; thereby they absorb certain per- 
ceptions about US policy which are commonly held by 
Americans concerned with foreign policy. On the other 
hand they are rarely privy to the thinking of senior 
policymakers and therefore do not have an accurate 
picture of the options that are or could be considered. 
Thus their analyses can be based on inadequate under- 
standing and will be read by people whose frame of 
reference is different from theirs. It would be 
easy to argue that policymakers should tell intelligence 
analysts what they think are viable options, but this 
line of argument is pointless. First, because policy- 
makers often don't know well enough themselves to be 
able to articulate the range of options 
any better than intelligence officers could. 

Second, they are often too busy to do so even if they 
could articulate options better than we. And finally 
because, for a variety of reasons, this is a matter outside 
the control of intelligence officers. Therefore, as a practical 
matter, what intelligence officers can do is to establish 
a rule that their work will contain an explicit statement 
of what they assume reasonable US policy options are 

and then to trace out the impact of those options on 

the situation under study. It may well be that the 

result will be a conclusion that US policy will not 

impact on unfolding events. However, it is also 

possible that the explicit statement of such assumptions 
will lead to further questioning of the analysts by 
policymakers who see other options that should be 
explored. Laying out options and exploring their 
implications need not imply endorsement of one or 

another option. 


Definitions also take on the character of assumptions. 
For example, political analysts typically (and subconsciously) 
define as equivalent the names of a government leader, 
government building, capital, country and nationality. 
Thus Brezhnev, Kremlin, Moscow, USSR, Russians and 
Soviets are used as synonymous. Such shorthand is an 
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insidious perpetrator of the homogeneity syndrome, 

j.e. treating foreign nations as if they were homogeneous. 
The syndrome often also involves the unwritten assumption 
that heads of state and leadership elites make decisions 
solely on the basis of a careful and completely rational 
calculus of world politico-strategic facts rather than 
emotion or internal power struggles. 

To root out this form of implicit assumption analysts 
should be forbidden for one full year to employ this 
shorthand. Furthermore until examination of internal 
diversity becomes commonplace a statement like, 

"T have examined all important groups in Brazil and 

find unanimity with respect to this matter." should be 
required in all analyses that do not explicitly 

examine the diversity in the situation under study. 


Consistency of conclusions from year to year is 
not a virtue, but consistency of assumptions among 
hypotheses within a particular analysis is essential. 
I cannot illustrate this point directly because those 
analyses I know which lay out hypotheses explicitly 
seem to keep assumptions consistent but most intelligence 
analyses don't lay out hypotheses and so the question 
of consistent assumptions doesn't arise. However, 
the flavor of the problem can be captured by looking 
at analyses of different questions pertaining to the 
same issue. For example; how often upon probing do we find the 
assumptions behind analyses of India's foreign policy 
to be different than the assumptions behind analyses --on 
the self-same issues--of Pakistan's foreign policy? 
All too often if one believes that Pakistan-India 
relations are a crucial part of the two nations’ 
foreign policies. 


Hypotheses. An analysis should explicitly 
evaluate at least three viable alternative hypotheses each 
of which purports to explain what we see. "Viable" means that a 
patently false strawman, set up only to be knocked down, 
won't do. If there is only one viable hypothesis the 
analyst should abort his project and turn to a worth- 
while topic. "Alternative hypotheses" should be 
meaningful in the sense that which hypothesis is true 
should have a significant impact on the future. 
Otherwise the matter is not worth pursuing. 


Setting three as the minimum number of 
hypotheses reflects the experience of people who haye 
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tried to explore decisionmaking in groups of people. 
The rule of thumb that has emerged from working with 
committees and decision papers is that 3-5 alternatives 
represents the range that can be handled when spoken 
and written English is the main analytic tool. Fewer 
than three alternatives rarely can cover the range 

of options credibly, and as one moves beyond five 
alternatives people tend to become 

confused. In the abstract, however, our goal should 
be to examine alternatives which for 

practical purposes exhaust the possibilities and yet 
are limited to those that are significantly different 
from one another in the sense that whether one 
hypothesis or another is true matters. In some OSI, 
OWI, OSR and OER analyses we already regularly 

employ many hypotheses, but these are cases where 

the main analytic tools are mathematical. To the 
extent that we improve our analysis in general we can 
perhaps handle more hypotheses. 


But for the moment we should walk with three 
and defer consideration of running. Lest we let our 
ambitions soar too high at once, we should note that 
the high, low and best estimates of foreign military 
forces produced in various intelligence documents 
exemplify one way to cope with the problem of 
alternative hypotheses. Anyone who doubts the 
difficulty of changing habits of doing business 
should retrace the agonies of the mid-60s when 
senior Defense Department policymakers dragged the 
Intelligence Community kicking and screaming into 
this now commonplace practice. It will be no easier 
to get analysts to formulate alternative political 
or economic scenarios, alternative motivational 
explanations, etc. 


This example, however, does serve to illustrate 
the practical importance of the use of alternative 
hypotheses because the projection of high, low and 
"best estimate" forces is in large part a matter of 
differing hypotheses of foreign intentions. The 
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intentions vs. capabilities debate frequently arises 
in military intelligence. Clearly intentions 
unsupported by capabilities are not important, 
except perhaps in the case of a foreign government 
which is so oblivious to consequences that it 

might act in defiance of the discrepancy between 

its intentions and capabilities. On the other hand 
focusing totally on capabilities (which is generally 
equivalent to focusing on "high" projections ) 
violates a central purpose of intelligence, to 
approximate the truth. The use of three 

hypotheses leaves a lot to be desired, but it is 

a better approximation than any one hypothesis. 


A final comment on hypotheses is that even 
in the 3-5 range, the "right" number of hypotheses 
to be subjected to detailed examination will vary 
with circumstances. Initially various hypotheses 
must be subjected to an informal evaluation 
in which the probability of each hypothesis 
is weighed with its importance if correct. In this 
way both very interesting but highly improbable 
and likely but trivial hypotheses will be eliminated 
from further consideration in order to increase the 
efficiency with which a smaller group of operationally 
important and reasonably likely hypotheses can be 
studied. 


Facts. A common saw, especially at middle 
levels, on the policy side of the government is, 
"Give me the facts, I'll draw my own conclusions." 
This naive view fails to realize that what is a fact 
is a matter of judgment, but a similar naivete often 
afflicts intelligence officers. The intelligence 
officer's version of this error is failure to 
realize that facts and theory are not separable. 

The accuracy of one's mental model of the process 
under study determines to a large extent whether 
one recognizes a fact when one sees it. To 
illustrate the point imagine ourselves as Eskimo 
schoolchildren reading a Post "Style" section 
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"Are we sure we were looking hard enough for 
evidence of a deal to buy the Soviet military's 
support of a move to depose Krushchev that we 
would not have missed a piece of evidence?" 


Eliminating irrelevant facts in order to 
economize effort is also important. Because 
connections are especially fuzzy, political 
analysis is particularly susceptible to the 
intrusion of irrelevant information. The 
basic guideline for this is that a piece of data 
is not a "fact" (worthy of attention) unless it 
serves to: 


-establish whether or not the mental 
model in use is adequate, 


-determine if the hypotheses under study 
are mutually exclusive and exhaustive, or 


-distinguish among the hypotheses under study. 


Facts should be presented reasonably consistently. 
Apples are rarely compared to oranges in OWI and such 
comparisons are reasonably easy to detect in OSR, 
but in OPA it is sometimes hard to decide what's an 
apple and what's an orange. 


Earlier we noted that intelligence analysis is 
unique in the degree to which it deals with 
information that belongs to knowledgeable thinking 
people who have reason to conceal or distort it. 
Therefore, we asserted that intelligence analysts 
must treat their raw material with a skepticism just 
short of paranoia. This applies particularly to 
those classes of "facts" which are susceptible to: 


-Shifts of meaning in transmission and 
translation, a problem that can be 
especially significant in dealing with 
human reporting. 
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-Deliberate falsification with intent to 
deceive. The history of the intelligence 
business is full of examples of this. 


-Fabrications without malicious intent, e.g. 
a common practice in journalism is to 
report as fact that which seems plausible 
in light of other information, ethnocentricity 
can be especially damaging here. This is 
not to argue that press reports should be 
treated with particular skepticism. It is 
to argue that the intelligence analyst 
should always be questioning the motives, 
knowledge and proclivities of every source 
of facts. 


Reasoning. The lines of reasoning in an 
analysis should be arranged so that a literate person 
can trace clearly the connections between facts, 
assumptions and conclusions for each hypothesis. 
Each step of the reasoning should be clearly 
identifiable as implication or intuition. There 
is no sin in the latter. Indeed, as long as it 
is labeled as such that is what we want to pay area 
specialists to be able to do. One of the great 
intellectual beauties of good analysis in any field 
is the use of innovative techniques to carefully 
reason across, to bridge, the unknowns in the 
problem. In this respect, the intelligence 
analyst's problem is not radically different 
from the archaeologist's. However, the praise 
of intuition and imagination in any field should 
not be a warrant for sloppy reasoning. Again at 
the level of detail appropriate to distinguishing 
among the hypotheses under study it should be 
possible to follow the analyst's thought processes 
and to critique them. The earlier discussion of 
the utility of models for these purposes applies 
to reasoning as well and need not be repeated. 
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Conclusions. Six years ago eee 
analysts that the least important element in rafting 
an estimate is the conclusions. This remains true 
although [[_]might not agree with the reasons stated 
here. If everything else in an analysis is done well 
the conclusions will follow relatively easily as long 
as we recognize that "conclusions" is plural. The 
most important part of presenting conclusions is 
developing this plurality. Doing so consists of 
some or all of the following points. 


Differing views of truth can be held by 
reasonable men who examine the same analysis. These 
differing views should be presented clearly and 
objectively. It is not sufficient to say that CIA 
holds one view and DIA another. The analysis must 
be retraced to show how different weights put on 
different assumptions and facts and how different 
reasoning lead to the different assessments of the 
debated hypotheses. 


A part of the conclusions of an analysis is 
identifying uncertainties and evaluating their 
importance, a process usually called sensitivity 
analysis. Virtually everything that enters into 
an intelligence analysis is uncertain so as a 
practical matter one cannot agonize over every 
uncertainty. The key in sensitivity analysis is 
to select those factors in the overall analysis 
which are both uncertain enough and influence 
the determination of which hypothesis is correct 
enough to merit detailed examination. Having 
made our selection we must ask how the evaluation 
of the hypotheses would change as the factors 
themselves change individually and in combination. 
There is no place in analysis that the intelligence 
officer's understanding of his subject is put to 
a harder test than in this elucidation. It is so 
hard that sins of omission must surely be more 
forgivable here than anywhere else. There are 
no rules except perhaps the thesis that it is 
better to be roughly right than precisely wrong. 
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Having dwelled on the need to illuminate 
uncertainty and disagreement, we must recognize 
that at the end we have a moral obligation to 
provide the policymaker with our judgment. This 
is an awesome responsibility because we are not 
dealing with trivia. One refuge from this respon- 
sibility has been sought all too often in the past, 
that is to state Delphic conclusions which can 
never be disproved by events. The obvious solution 
of requiring the analyst or team to state cate- 
gorically which hypothesis is true is not, however, 
workable because analysts simply do not (and 
cannot) know. 


A reasonable (and workable although painful ) 
compromise is to require stating the likelihood 
of each hypothesis being true. This can be done 
with at least three degrees of precision. 


-Approximate probabilities. Recognizing 
that precise probability statements, say 

to the second decimal place like 93%, will 
normally not be worth the effort to produce, 
we should only aspire at most to 

approximate probability statements like 

90% or 35%. 


-Odds. One can give a somewhat less 
precise assessment by using odds based on 
whole numbers. A mathematician 
would say that this means little in 
principle. However, some people feel 
more comfortable saying 
"3 out of 4 chances"than "75% probability," 
and for whatever reasons, readers do 
seem to take statements like “even odds" 
or "50-50 chance" as being less precise 
than "50% probability." 
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-Adjectival statements. For 

analysts who are simply petrified with 
fear at the prospect of using numbers 
words must suffice. The Sherman Kent 
chart is an attempt to codify such 
statements and to relate them to more 
precise measures. If we do no more than 
agree to use some such standard 
terminology we will have made progress 
because now an adjective like, for example, 
"probable" can mean anything a 

"just better than equally ey 
"virtually certain.' 


CLOSING REMARKS 


When one has laid out an analysis in accordance 
with these guidelines what does one have? Some very long and 
pretty dull reading to be sure. However, one has 
the basis for a reasonably competent writer (who 
needs to know little about the subject) to repackage 
the product ad infinitum to serve the needs of 
high-level and Tow-level policymakers, and general 
thrill seekers. If the analysis is properly done, 
packaging the product for dissemination is relatively 
easy. Our current agonies over repeated reviews 
and debates ad nauseam over turns of phraseology 
arise in part because we are papering over obvious 
cracks in our analytic walls. 


To repeat an assertion made in the context of 
models, one also has a vehicle which will facilitate 
future critiques of the analysis for everything is 
laid out naked inviting any would-be kibitzer to 
take pot shots. Even folk who do not understand the area under 
study will be able to detect the standard errors 
that have plagued men's intellectual endeavors in 
many fields. For example “good analysis," as seen 
here, makes such fallacies as "Achieving maximum 
gain at minimum cost" stand out more starkly 
to would-be critics. 
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Similarly, critics are invited to proclaim 
that the analyst has used too few indicators. 
Why will this be the case? Because people 
always use too few indicators whether they're 
betting on horses, planning personal estates or 
trying to detect the outbreak of war. Unconsciously 
we all pick a few, no more than half a dozen, things 
we look at in determining trends. This is inherent 
in our mental machinery. The analytic process 
can extend our horizons to perhaps tens or hundreds 
of things, depending upon how we go about it. 


This assertion about the role of good analysis 
in facilitating criticism is repeated 
because it is fundamental. We must produce work that 
can be criticized! This goes very much against the 
human grain. No one likes to be "torn apart" by 
critics; everyone subconsciously seeks to avoid it 
by a variety of devices, some of which have been 
discussed above. Accepting this view implies 
that we will shift our thinking about good 
analysis away from the finished products of the 
Intelligence Community toward the intermediate 
products and processes. However, without a view of "good 
analysis" similar to the view here we have no way, 
consistently year after year as people come and go, 
to build a better and better understanding. 


We should, however, end on a humble note. 
The Earl of Halsbury, drawing on his experience 
in World War II, lectured a group of analysts 
twenty-five years ago and pointed out that we 
will never reach the end of the process. He said, 


"The content of a human social or 
historical situation is always richer than 
verbal analysis can display. Reality is like 
a tangled skein of threads. With infinite 
patience the analyst dissects one thread 
out of the bundle and is delighted to observe 
a causal nexus between its parts. We thus 
reach truth, but never The Truth. The 
reality inevitably bristles with more 
detail than can be apprehended as a whole." 
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